Method and system for machine learning classification based on structure or material segmentation in an image

ABSTRACT

A system and method for classifying a structure or material in an image of a subject. The system comprises: a segmenter configured to form one or more segmentations of a structure or material in an image and generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores.

FIELD OF THE INVENTION

The present invention relates to a computer vision system and method,employing machine learning and in particular deep neural networks, forclassifying (and monitoring changes in) an image (such as a medicalimage), based on structural or material segmentation. Possible medicalimaging applications include Computed Tomography (CT), MagneticResonance (MR), Ultrasound, HRpQCT (High-Resolution peripheralQuantitative Computed Tomography), and Pathology Scanner imaging.

BACKGROUND

Computer vision and image processing techniques have been applied tomedical image analysis. Some computer-aided systems achieve the analysiswith two steps: segmentation and quantitative calculation. Segmentationis the process of segmenting (or differentiating) structures or objectsin an image, such as a medical image, from one another bydifferentiating pixels (in 2D image) or voxels (in 3D image) in theimage. Based on the segmentation, quantitative features such as volume,shape, and density are calculated. For example, lesion size and shapeare calculated after the lesion has been segmented in a brain CT or MRIscan; the bone mineral density may be calculated after the femoral neckis segmented in a hip DXA (dual-energy x-ray absorptiometry) scan. Adoctor may make a diagnosis or treatment decision after he or she hascompared such calculated values with healthy reference data.

For example, a T-score is the standard score of a patient's bone mineraldensity compared to the young normal reference mean. The WHO (WorldHealth Organization) defines osteoporosis as a T-score of −2.5 or lower,that is, a bone density that is two and a half standard deviations ormore below the mean of a 30-year-old healthy man/woman.

The segmentation may be achieved manually, semi-manually, orautomatically. In an example of manual segmentation, a user operates acomputer to move a rectangular box over a hip DXA scan and therebyselect the region of the femoral neck.

Semi-manual segmentation may be performed by an image processing programemploying a user's initialisation or input. For example, a user mayoperate a computer to draw an approximate bone boundary on a wrist CTscan; the program then adjusts the approximate boundary into a contourthat segments bone from the surrounding tissues.

Automatic segmentation may be performed by utilizing the features of theobject of interest, such as intensity values, edges and shapes. In oneexisting example, a voxel-value based thresholding method is used tosegment bone from the surrounding tissues in CT scans. Some otherprograms use machine learning algorithms to train a classifier tosegment abnormal tissues in medical images. For example, a feature-basedmachine learning algorithm, such as a support vector machine and adecision tree, may be used as a classifier by using tumour images andnormal images as training data. The trained classifier slides throughthe new image “window” by “window” to segment any image regions oftumour tissues.

Machine learning algorithms have shown promising accuracy and efficiencyin this field. However, it is a significant challenge to both collectsufficient training data and to annotate the training data. The trainingimages must be annotated by experts, which is a tedious andtime-consuming process. Moreover, in some applications, it may be verydifficult or nearly impossible to accurately annotate the trainingimages, even for experts. For example, in bone quality assessment, atransitional zone exists at any sample composed of both cortical andtrabecular bones. The transitional zone comprises the inner cortexadjacent to the medullary canal and trabeculae abutting against thecortex contiguous with the endocortical surface. The transitional zoneis a site of vigorous bone remodelling. It is important to identify andsegment this region in bone microstructure assessment but, owing tolimitations in image resolution, it is essentially impossible for anexpert to annotate this region both accurately and consistently. Withoutannotated images as training data, the segmentation model cannot betrained.

In the last few years, deep learning or deep neural networks haveoutperformed human in many visual recognition tasks such as naturalimage classification. In an exemplary CNN (Convolutional Neural Network)implementation, the network comprises input layer, hidden layers, and anoutput layer. An image is fed into the network through the input layer.The image is sampled and applied with convolutional operations togenerate hidden layers. The output of each layer is used as input to thenext layer in the network. The output layer is fully connected at theend that will output a classification result. Training data are imageswith classification labels. The training process obtains the parametersof the neural network. After the training is finished, a new image willbe processed by the neural network with the obtained parameters togenerate a classification result. For example, a deep neural networkalgorithm may be used to train a model to determine the condition (forexample, no, mild, moderate, severe) of diabetic retinopathy from OCT(Optical Coherence Tomography) images.

However, this end-to-end solution brings two problems in clinicalpractices. First, the end-to-end solution is a black box: the input isthe medical image, and the output the classification of diseases orconditions. It is difficult to interpret the process whereby the neuralnetwork makes its decision—so it is difficult for the user to assess thereliability of the classification results. Secondly, this solutionrequires a substantial amount of training data. As discussed above, inmedical applications annotating or labelling the training data is atedious and time-consuming process. Collecting enough training data foreach category of each type of classification result thus poses asignificant challenge.

SUMMARY

According to a first aspect of the invention, there is provided a systemfor classifying a structure or material in an image of a subject,comprising:

-   -   a segmenter configured to form one or more segmentations of a        structure or material in an image (comprising, for example, a        medical image) and generate from the segmentations one or more        segmentation maps of the image including categorizations of        pixels or voxels of the segmentation maps assigned from one or        more respective predefined sets of categories;    -   a classifier that implements a classification machine learning        model configured to generate, based on the segmentations maps,        one or more classifications and to assign to the classifications        respective scores indicative of a likelihood that the structure        or material, or the subject, falls into the respective        classifications; and    -   an output for outputting a result indicative of the        classifications and scores.

In an embodiment, the classifier generates the one or moreclassifications based on the segmentations maps and non-image datapertaining to the subject.

The system may be configured to train the classification machinelearning model.

In an embodiment, the segmenter comprises:

-   -   i) a structure segmenter configured to generate structure        segmentation maps including categorizations of the pixels or        voxels assigned from a predefined set of structure categories,    -   ii) a material segmenter configured to generate material        segmentation maps including categorizations of the pixels or        voxels assigned from a predefined set of material categories,        and/or    -   iii) an abnormality segmenter configured to generate abnormality        segmentation maps including categorizations of the pixels or        voxels assigned from a predefined set of abnormality or        normality categories.

In an example, the structure segmenter is configured to employ astructure segmentation machine learning model to generate the structuresegmentation maps, the material segmenter is configured to employ amaterial segmentation machine learning model to generate the materialsegmentation maps, and the abnormality segmenter is configured to employan abnormality segmentation model to generate the abnormalitysegmentation maps. The structure segmenter may be configured to trainthe structure segmentation machine learning model, the materialsegmenter to train the material segmentation machine learning model,and/or the abnormality segmenter to train the abnormality segmentationmodel.

In an embodiment, the system further comprises a segmentation mapprocessor configured to process the segmentation maps before thesegmentation maps are input by the classifier. In an example, thesegmentation map processor is configured to down-sample the segmentationmaps.

In an embodiment, the classification machine learning model comprises aneural network, a support vector machine, a decision tree, or acombination thereof. For example, the classification machine learningmodel may comprise a neural network that includes convolutional neuralnetwork layers and fully-connected neural network layers.

In an embodiment, the image is a medical image, and the classificationscorrespond to probabilities that the structure or material, or thesubject, will sustain a specified condition or symptom in respectivetimeframes. On an example, the timeframes include a shorter-termtimeframe, a longer-term timeframe, and at least one intermediate-termtimeframe intermediate the shorter-term timeframe and the longer-termtimeframe. In another example, the condition or symptom is bonefracture.

In an embodiment, the image is a medical image, and the classificationscorrespond to probabilities that the structure or material, or thesubject, will sustain respective conditions or symptoms. In an example,the conditions or symptoms are bone conditions.

In an embodiment, the image is a medical image, and the classificationscorrespond to probabilities of respective rates of disease or pathologyprogression. For example, the classifications may compriseclassifications corresponding any one or more of: stable, modestdeterioration, and accelerated deterioration.

In an embodiment, the image is a medical image, and the classificationscorrespond to probabilities of efficacy of respective treatment options.For example, the treatment options may include an antiresorptivetreatment and/or an anabolic treatment.

In an embodiment, the image is a medical image, and the classificationscorrespond to respective medical conditions. For example, the medicalconditions may include any one or more of: osteomalacia, tumour,osteonecrosis and infection.

In an embodiment, the classification machine learning model is a modeltrained with image data and non-image data relating to trainingsubjects, and generates the respective scores based on image data(typically constituting one or more images) and non-image data relatingto the subject.

According to a second aspect of the invention, there is provided acomputer-implemented method for classifying a structure or material inan image of a subject, comprising:

-   -   forming one or more segmentations of a structure or material in        an image;    -   generating from the segmentations one or more segmentation maps        of the image including categorizations of pixels or voxels of        the segmentation maps assigned from respective predefined sets        of categories of the structure or material;    -   using a classification machine learning model to generate, based        on the segmentations maps, one or more classifications and to        assign to the classifications respective scores indicative of a        likelihood that the structure or material, or the subject, falls        into the respective classifications; and    -   outputting a result indicative of the classifications and        scores.

In an embodiment, the classification machine learning model is used togenerate the one or more classifications based on the segmentations mapsand non-image data pertaining to the subject.

The method may include training the classification machine learningmodel.

In an embodiment, forming the one or more segmentations comprises:

-   -   i) generating structure segmentation maps including        categorizations of the pixels or voxels assigned from a        predefined set of structure categories,    -   ii) generating material segmentation maps including        categorizations of the pixels or voxels assigned from a        predefined set of material categories, and/or    -   iii) generating abnormality segmentation maps including        categorizations of the pixels or voxels assigned from a        predefined set of abnormality or normality categories.

For example, the method may include employing a structure segmentationmachine learning model to generate the structure segmentation maps, amaterial segmentation machine learning model to generate the materialsegmentation maps, and an abnormality segmentation model to generate theabnormality segmentation maps. In particular, the method may includetraining the structure segmentation machine learning model, the materialsegmentation machine learning model, and/or the abnormality segmentationmodel.

According to a third aspect of the invention, there is provided acomputer-implemented diagnostic method, comprising the method of thesecond aspect.

According to a fourth aspect of the invention, there is provided acomputer-implemented method for training a classification machinelearning model for classifying a structure or material in an image of asubject, the method comprising:

-   -   dividing annotated segmentation maps and annotated non-image        data into a training set and a testing set (such that, as a        result, the training set and the testing each include some        annotated segmentation maps and some annotated non-image data),        the annotated segmentation maps obtained by segmenting one or        more images;    -   implementing a classification machine learning model, including        initializing parameters of the classification machine learning        model;    -   updating the parameters of the classification machine learning        model by running a learning algorithm on the training data;    -   testing the classification machine learning model on the testing        data;    -   evaluating whether the classification machine learning model has        satisfactory performance; and    -   when the performance is found to be satisfactory, outputting the        classification machine learning model for deployment or flagging        the classification machine learning model as fit for deployment.

This aspect may be used in conjunction or in combination with (or as apart of) the second aspect, such as to train the classification machinelearning model of the second aspect.

The method may include segmenting the one or more images (such as in thecourse of generating the annotated segmentation maps).

In an embodiment, the method includes, when the performance is found tobe unsatisfactory, collecting more image and non-image data for trainingthe classification machine learning model.

The classification model can be trained by various machine learningalgorithms, so may comprise—for example—a neural network, a supportvector machine, a decision tree, or a combination thereof.

Thus, in one embodiment, the classification machine learning modelcomprises a neural network having a plurality of layers comprisingartificial neurons, wherein the parameters comprise layer numbers,neuron numbers, neuron weights, and neuron function parameters; andtesting the classification machine learning model includes testing theclassification machine learning model on the testing data.

In an embodiment, updating the parameters includes determining agradient of a loss function.

In an embodiment, the images are medical images and the non-image datacomprise clinical records.

In an embodiment, the method includes dividing the annotatedsegmentation maps and the annotated non-image data into the trainingset, a development set and the testing set, and using the developmentdata to investigate the learning procedure and to tune the parameters(and, when the classification machine learning model comprises a neuralnetwork, tune the layers).

According to a fifth aspect of the invention, there is provided acomputer program comprising program code configured, when executed byone of more computing devices, to implemented the method of any one ormore of the second to fourth aspects. According to this aspect, there isalso provided a computer-readable medium, comprising such a computerprogram.

It should be noted that any of the various individual features of eachof the above aspects of the invention, and any of the various individualfeatures of the embodiments described herein including in the claims,can be combined as suitable and desired.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the invention may be more clearly ascertained, embodimentswill now be described by way of example with reference to the followingdrawing, in which:

FIG. 1 is a schematic view of a classification system according to anembodiment of the present invention;

FIG. 2 is a high-level schematic diagram illustrating the operation ofthe classification system of FIG. 1, in calculating bone fracture riskof a subject from a medical imaging scan;

FIG. 3 is a schematic view of the operation of the segmenter of theclassification system of FIG. 1;

FIG. 4 is a schematic illustration of the operation of classifier of theclassification system of FIG. 1;

FIG. 5 is a flow diagram of the training of the classification neuralnetwork of the classification system of FIG. 1.

DETAILED DESCRIPTION

FIG. 1 is a schematic view of a classification system 10 for classifyinga structure or material in a medical image (based on structural andmaterial segmentation), according to an embodiment of the presentinvention.

Referring to FIG. 1, system 10 comprises a classification controller 12and a user interface 14 (including a GUI 16). User interface 14 isprovided for representing information to a user and for receiving input(including feedback) from a user; it typically comprises one or moredisplays (on one or more of which may be displayed the GUI 16), a webbrowser, a keyboard and a mouse, and optionally a printer.Classification controller 12 includes at least one processor 18 and amemory 20. System 10 may be implemented, for example, as a combinationof software and hardware on a computer (such as a personal computer ormobile computing device), or as a dedicated image segmentation system.System 10 may optionally be distributed; for example, some or all of thecomponents of memory 20 may be located remotely from processor 18; userinterface 14 may be located remotely from memory 20 and/or fromprocessor 18. For example, system 10 may be implemented in aservice-oriented architecture, with its components communicating witheach other over a communication network such as a LAN (local areanetwork), WAN (wide area network) or the internet System 10 may bedeployed in the cloud, and its use shared by users at differentlocations.

In certain other embodiments, system 10 is implemented as a standalonesystem (of software and hardware) or as standalone software executableby a computer, and deployed in one location; for example, system 10 maybe deployed in a hospital, medical clinic or other clinical setting.

Memory 20 is in data communication with processor 18, and typicallycomprises both volatile and non-volatile memory (and may include morethan one of each type of memory), including RAM (Random Access Memory),ROM and one or more mass storage devices.

As is discussed in greater detail below, processor 18 includes asegmenter 22 (which includes a structure segmenter 24 a, a materialsegmenter 24 b, and an abnormal segmenter in the form of an abnormalmaterial segmenter 24 c), an segmentation map processor 26, and anon-image data processor 28. Processor 18 also includes a classifier 30,an I/O interface 32 and a results output 34.

Memory 20 includes program code 36, image data 38, non-image data 40,segmentation models 42 (including, in this example, structuresegmentation models 44, material segmentation models 46, and abnormalitysegmentation models in the form of abnormal material segmentation models48), segmentation maps 50 (including, in this example, structuresegmentation maps 52, material segmentation maps 54, and abnormalitysegmentation maps in the form of abnormal material segmentation maps56). Structure segmenter 24 a, material segmenter 24 b and abnormalmaterial segmenter 24 c train the respective segmentation models 44, 46,48, and use segmentation models 44, 46, 48 to perform segmentation onincoming images and generate structure segmentation maps 52, materialsegmentation maps 54, and abnormal material segmentation maps 56,respectively.

Memory 20 also includes a classification machine learning model in theform of a classification neural network 58, which is trained and used byclassifier 30 to perform classification by using segmentation maps 50and non-image data 40. Classification controller 12 is implemented, atleast in part (and in some embodiments entirely), by processor 18executing program code 36 from memory 20.

It should be noted that, as the present embodiment relates to theclassifying of structures and/or materials in a medical image, abnormalmaterial segmenter 24 c may also be referred to as an abnormal tissuesegmenter, and abnormal material segmentation maps 56 may also bereferred to as abnormal tissue segmentation maps.

In broad terms, I/O interface 32 is configured to read or receivemedical image data and non-image data pertaining to a subject, and tostore these data as image data 38 and non-image data 40 of memory 20 forprocessing. Image data 38 is typically in the form, in this embodiment,of a medical image of—for example—a region of the body of a subjectNon-image data 40 typically includes subject or patient information fromvarious structured and non-structured data sources, collected throughouta subject's medical consultations, treatment and follow-upconsultations. Subject structured data may include basic subjectinformation such as sex, age, weight, height; laboratory test resultssuch as blood test results and DNA test results; treatment data such asthe types of medication and dosage; and questionnaire data such assmoking and drinking habits and fracture history. Subject unstructureddata may include text documents of laboratory results, doctors' notes,and radiological reports. Non-image data 40 may in a variety of formats,such as numerical data, text, voice, and video.

Segmenter 22 processes image data 38 (constituting one or more medicalimages) and uses structure segmentation models 44, material segmentationmodels 46 and abnormal material segmentation models 48 to generate—fromimage data 38—structure segmentation maps 52, material segmentation maps54 and abnormal material segmentation maps 56, respectively, whichcharacterize image data 38 in different ways. Classifier 30 then inputsthe resulting segmentation maps 50 and non-image data 40, and generatestherefrom results in the form of a classification output. Theclassification output is, in this embodiment, presented to users or usedfor further analysis via I/O interface 32 and at either results output34 and/or user interface 14.

The classification output of classifier 30 (in this embodiment,generated using classification neural network 58) comprises a respectivecondition score for each of one or more classifications (and preferablyfor each of a plurality of possible classifications). Each scorerepresents a predicted likelihood that the subject falls into thecorresponding classification. In the present example of bone fragilityassessment, the classifications are “negligible fracture risk”,“imminent fracture risk”, “intermediate-term fracture risk”, and“long-term fracture risk”. The classification output is described inmore detail below.

In an alternative embodiment, the classifier outputs a respectivedisease progression score for each of one or more condition progressionstates. Each score represents a predicted likelihood that a currentcondition will progress to another condition. For example, in bonefragility assessment, the disease progressions may include “stable”,“modest deterioration”, and “accelerated deterioration”.

In still another embodiment, the classifier outputs a respectivetreatment score for each of multiple treatment options. Each scorerepresents a predicted likelihood that the treatment is the mostefficient for the patient. For example, in a bone fragility assessment,the treatment options may include “antiresorptive”, “anabolic”, and“antiresorptive+anabolic”.

In a further embodiment, the classification output comprises a score foreach of one or more possible classifications corresponding to knownmedical conditions or pathologies. For example, in a bone fragilityassessment, these classifications could be “osteomalacia”, “tumour”,“osteonecrosis” and “infection”. In such an embodiment, the resultingscores represent the degree to which the (e.g. bone) sample of thesubject conforms to that classification/condition. If only oneclassification has a significant score, or one classification has ascore that is significantly greater than all other scores, thatclassification may be regarded as a diagnosis, or suggested diagnosis,of the corresponding condition or pathology.

In certain embodiments, the classification output comprises two or moresets of such scores (selected from the aforementioned examples orotherwise).

Returning to FIG. 1, as will be appreciated by the skilled person inthis art, image data 38—constituting one or more medial images—comprisesdata generated by one or more of a variety of medical image modalities(such as HRpQCT, or High-Resolution peripheral Quantitative ComputedTomography) implemented by one or more medical imaging devices (such asa HRpQCT scanner). Each of these devices scans a sample (whether in vivoor in vitro) and creates a visual representation, generally of a portionof the interior of a subject's body. The medical images may depict, forexample, a part of a body or a whole body of a subject (e.g. the brain,the hip or the wrist). The medical images might be acquired by scanningthe same sample or body part using different imaging modalities, asdifferent imaging modalities may reveal different characteristics of thesame sample or body part. The medical images might be acquired byscanning different body parts using the same image modalities, asdifferent body parts of the same patients might provide differentinsights towards a better diagnosis of diseases or conditions. Forexample, in bone fragility assessment, both the wrist and the leg of apatient may be scanned by an HRpQCT scanner (or indeed acquired byscanning the different samples or body parts using different imagingmodalities) to provide information for use in assessing a subject's bonequality.

The image data 38 may constitute a 2D (two-dimensional) image that maybe represented as a 2D array of pixels, or a 3D (three-dimensional)image that may be represented as a 3D array of voxels. For convenience,the medical images described below are 3D images that may be representedas a 3D array of voxels.

As mentioned above, the one or more received medical images, stored inimage data 38, are segmented by segmenter 22, using trained segmentationmodels 42, into respective segmentation maps 50. Each segmentation map52, 54, 56 characterizes the respective medical image differently. Astructure segmentation map 52 represents the medical image as one ormore different anatomical structures from a predefined set ofstructures. For example, a wrist CT scan may be segmented into compactcortex, transitional zone, and trabecular region. Material segmentationmap represents the medical image into multiple different materials froma predefined set of materials. For example, a wrist CT scan might besegmented into mineralized material, fully mineralized material, redmarrow in the trabecular region, and yellow marrow in the trabecularregion. An abnormal material segmentation map 56 represents the medicalimage as normal material and abnormal material (or, in this example,normal tissue and abnormal tissue). For example, a tumour or fracturemight be segmented from a wrist CT scan and represented in an abnormalmaterial segmentation map 56 as ‘abnormal’.

Segmentation maps 50 are inputted into classifier 30, in combinationwith non-image data 40. Classifier 30 generates one or moreclassification outputs based on segmentation maps 50 and the non-imagedata 40. Input data of classifier 30 is generally multi-dimensional, soclassifier 30 is implemented with machine learning algorithms, such as aneural network, support vector machine, decision tree, or a combinationthereof.

In this embodiment, classifier 30 employs or is implemented asclassification neural network 58 (though in other embodiments, othermachine learning algorithms may be acceptable), including—in thisexample—convolutional neural network layers and fully-connected neuralnetwork layers. Classification neural network 58 is trained withtraining data, as is described below.

As mentioned above, the ultimate classification output is outputted bysystem 10 to a user via results output 34 or user interface 14. Theclassification output may optionally include a visual presentation ofone or more of the corresponding segmentation maps 50. Segmentation maps50 may be presented in case they can assist a user in interpreting theclassification output, such as in assessing the reliability of theresults.

FIG. 2 is a high-level schematic diagram 50 illustrating the operationof system 10 in calculating bone fracture risk of a subject from amedical imaging scan, in this example a wrist HRpQCT scan 62 (also shownin negative at 62′). As shown in FIG. 2, system 10 receives the wristHRpQCT scan 62 comprising a plurality of slices. (As will be appreciatedby the skilled person, an HRpQCT scan can comprise 100 or more slices,but four slices are depicted in the figure for simplicity.)

Segmenter 22 segments scan 62 into a structure segmentation map 52 inwhich the scan is segmented into compact cortex, outer transitionalzone, inner transitional zone, and trabecular region. Segmenter 22segments scan 62 into material segmentation map 54, in which scan 62 issegmented into surrounding muscle, surrounding fat, yellow marrowadipose, and red marrow adipose. Data 64 comprising segmentation maps52, 54, abnormal material segmentation maps 56 and non-image data 40(e.g. clinical factors including sex and age) are processed by trainedclassifier 30 to generate classification outputs. The classificationoutputs include segmentation maps 52, 54 and a table or report 66. Tableor report 66 includes, in numerical and/or graphical form, fractureprobabilities in each category of fracture risk: imminent fracture risk68 a (fracture within two years: t<2 y), intermediate-term fracture risk68 b (fracture within two to five years: 2≤t<5 y), long-term fracturerisk 68 c (fracture in five to ten years, 5≤t≤10 y), and negligiblefracture risk 68 d. In the illustrated example, the probability that thewrist is at risk of fracture within two years is 95.6%, that the wristis at a risk of fracture in two to five years 2.4%, that the wrist is ata risk of fracture in five to 10 years 1.6%, and that the wrist is atnegligible risk of fracture is 0.3%. In other words, the probabilitythat the subject will not have a wrist fracture in the next five years(either because the wrist has negligible risk of fracture or becausethere is only a long-term fracture risk) is only 4.4%. Table or report66 does not include a diagnosis (e.g. that the subject hasosteoporosis), but it will be appreciated that these probabilities maybe of great value, including—for example—to prompt the subject to pursuea diagnosis, such as by undergoing medical examination or consultation.

FIG. 3 is a schematic view at 70 of the operation of segmenter 22.Segmenter 22 is configured to receive input includes one or more medicalimages (from image data 38) and to process the images so as to generatesegmentation maps 50. The medical images might be acquired using thesame imaging modality by scanning different body parts of a patient. Forexample, in some applications of assessing bone quality, both wrist andleg of a patient might be scanned by an HRpQCT scanner for theassessment. The medical images might be acquired using different imagingmodalities by scanning the same or different body parts of a patient.For example, in some other applications of assessing bone quality, boththe wrist HRpQCT scan and the hip DXA scan of a patient are acquired forthe assessment (though again bearing in mind that the medical images maybe acquired by scanning the different samples or body parts using otherimaging modalities).

Referring to FIG. 3, segmenter 22 implements one or more processingbranches 1 to n, (labelled 72 ¹, . . . , 72 ^(n) in the figure)corresponding to medical images 1 to n of the subject (labelled 38 ¹, .. . , 38 ^(n)). In the case of plural processing branches, medicalimages 38 ¹, . . . , 38 ^(n) may be due to—for example—different imagingmodalities (labelled 74 ¹, . . . , 74 ^(n)), as is the case in theillustrated example, different body parts, different scans of a singlebody part, or a combination two or more of these. Respectivesegmentation branches 72 ¹, . . . , 72 ^(n) are configured to receive animage, to segment the image according to image type (such as withdifferent program code 36), and to generate the branch output(comprising the segmentation maps 50 of the input image).

To process a respective input medical image, segmenter 22 is configuredfirst to select a processing branch of processing branches 1 to naccording to the type of the input image. Segmenter 22 ascertains thetype of image according to the sample (e.g. scanned body part) andimaging modality, information that can be determined from the respectiveimage, including from metadata stored in a header file of the medicalimage and/or from the file type. For example, the information of scannedbody part and imaging modality may be accessed from the metadata.

Each input medical image 1 to n is processed by one or more of threesegmentation programs (viz. structure segmenter 24 a, material segmenter24 b, and abnormal material segmenter 24 c) into the correspondingsegmentation maps 52, 54, 56. Segmenter 22 thus employs up to ninstances each of segmenters 24 a, 24 b, 24 c (labelled 24 a ¹, . . . ,24 a ^(n), 24 b ¹, . . . , 24 b ^(n), and 24 c ¹, . . . , 24 c ^(n),respectively), either in parallel or sequentially, though the number ofsuch instances of each segmenter 24 a, 24 b, 24 c (being from 0 to n ineach case) may differ.

Structure segmenter 24 a, material segmenter 24 b, and abnormal materialsegmenter 24 c may generate respective segmentation maps in eachprocessing branch 72 ¹, . . . , 72 ^(n). In FIG. 3, for example,structure segmenter 24 a, material segmenter 24 b, and abnormal materialsegmenter 24 c generate respective segmentation maps corresponding tomedical imaging modalities 1 to n; the resulting structure segmentationmaps, material segmentation maps and abnormal tissue segmentation mapsare correspondingly labelled structure segmentation maps 52 ¹, . . . ,52 ^(n), the material segmentation maps 54 ¹, . . . , 54 ^(n) and theabnormal tissue segmentation maps 56 ¹, . . . , 56 ^(n). It should benoted, however, that in some applications it may not be possible ordesirable to generate all three types of segmentation map. This may bedue, for example, to the limitations of the images, of the imagingmodalities, or of segmenters 24 a, 24 b, 24 c (arising, for example,from limitations in segmenter training data).

Structure segmenter 24 a, material segmenter 24 b, and abnormal materialsegmenter 24 c assign to each voxel of these segmentation maps 50 one ormore ‘types’ (or ‘categories’) from respective predetermined sets oftypes (or categories). Thus, in this embodiment structure segmenter 24 aassigns a respective structure type (from a predefined set of structuretypes) to each voxel in the scan. For example, a wrist HRpQCT scan issegmented into a structure segmentation map 52 in which each voxel inthe scan is assigned a structure type (or category) from the set of“surrounding tissues”, “compact cortex”, “transitional zone”, and“trabecular region”.

Material segmenter 24 b assigns a respective material type (from apredefined set of material types) to each voxel. For example, in thisembodiment, material segmenter 24 b segments a wrist HRpQCT scan into amaterial segmentation map 54 in which each voxel in the scan is assigneda material type from the set of “mineralised material”, “fullymineralised material”, “red marrow adipose”, and “yellow marrowadipose”.

Abnormal material segmenter 24 c assigns a respective abnormality ornormality type (from a predefined set of abnormalities or normalitytypes, such as a set comprising “normal” and “abnormal”) to each voxel.For example, in this embodiment, abnormal material segmenter 24 csegments a wrist HRpQCT scan into an abnormal tissue segmentation map 54in which each voxel in the scan is assigned either “normal” or“abnormal”. Optionally, in certain other embodiments, abnormal materialsegmenter 24 c can distinguish different types of abnormality, and thepredefined set of abnormality or normality types may include—in additionto “normal”—and one or more specific abnormalities particular to thesample type under examination; if the sample is bone, these may include,for example, “fracture crack” or “bone tumour”. In such an embodiment,the set may optionally include “abnormal” for cases in which abnormalmaterial segmenter 24 c cannot determine a specific type of abnormality.

In some implementations, structure segmenter 24 a, material segmenter 24b, and abnormal material segmenter 24 c assign respective types withconfidence limits or probabilities to each voxel in the medical image.In some other implementations, structure segmenter 24 a, materialsegmenter 24 b, and abnormal material segmenter 24 c may assign aplurality of types (each optionally with a confidence limit orprobability) to each voxel in the medical image. For example, structuresegmenter 24 a—when segmenting a wrist HRpQCT scan—may assign both“transitional zone” and “trabecular region” to ambiguous voxels, butwith respective (and typically different) confidence limits orprobabilities.

Segmenter 22 generates segmentation maps 50 by using the trainedsegmentation models 42 (including structure segmentation models,material segmentation models and abnormal material segmentation models).The segmentation models are trained using machine learning algorithms(such as a neural network, a support vector machine, a decision tree, ora combination thereof). In this embodiments, the segmentation models aretrained using deep neural networks that comprises multiple layersinclude convolutional neural network layers, fully connected neuralnetwork layers, normalisation layers, and multiplicative layers.

Segmenter 22 may also (or alternatively) perform segmentation usingnon-machine learning based methods, such as a method based on thelocation of edges, corners, and transitional slopes, or on globalfeatures such as histogram and intensity values of the image. Forexample, U.S. Patent Application Publication No. 2012/0232375 A1(“Method and System for Image Analysis of Selected Tissue Structures”)discloses a method for segmenting the transitional zone between thecompact cortical and trabecular region from a wrist CT scan, based onlocal and global features of a bone: in many applications, it would besuitable to implement this method in segmenter 22.

FIG. 4 is a schematic illustration of the operation 80 of classifier 30.Classifier 30 is configured to receive input that includes segmentationmaps 50 (generated by segmenter 22) and non-image subject data 40, andto process that input so as to generate one or more classificationresults.

However, both the segmentation maps 50 and non-image subject data 40 areprocessed before being passed to classifier 30 by segmentation mapprocessor 26 and non-image data processor 28, respectively. For example,in some implementations, it may be expedient to down-sample segmentationmaps 50 into lower resolution maps, such as to allow faster imageprocessing by classifier 30; such processing, if desired or required, isperformed by segmentation map processor 26. In some implementations,segmentation map processor 26 sets the type of any voxels (in aparticular segmentation map) that have been assigned more than one type(though typically with different confidence limits or probabilities),such as by assigning to the voxel the type that has the higher orhighest probability.

Non-image data may include structured and unstructured data. Non-imagedata processor 28 is configured to employ a variety of techniques toprocess any structured data by extracting features from it, in each caseaccording to the structure and form of the data. For example, structureddata are typically stored and maintained in structured data storage suchas database tables, .json files, .xml files and .csv files. Non-imagedata processor 28 extracts features from structured data by querying therequired parameters and attributes from the data's respective sources.

Non-image data processor 28 processes unstructured data in two steps:firstly by converting it into structured data, then by extractingfeatures from the converted data. The conversion method employed bynon-image data processor 28 is specific to each source. For example, toconvert a doctor's notes into structured data, non-image data processor28 employs a trained model of optical character recognition (OCR) toconvert the notes into text recognisable by a computer. Non-image dataprocessor 28 then parses the converted text using keywords such as, inthis example, “fractures”, “pain”, “fall”, etc. Once the unstructureddata has been converted into structured data, non-image data processor28 then extracts features from the now structured data.

The processed non-image data 40 and segmentation maps 22 are passed toclassifier 30, which uses these inputs to generate a classificationoutputs comprising classification score (such as disease conditionscores 82, disease progression scores 84, and/or treatment scores 86).

FIG. 5 is a flow diagram 90 of the training of classification neuralnetwork 58 of classifier 30 with a deep neural network, according to anembodiment of the present invention. As shown in the figure, at step 92,data—including medical image data and non-image data—are collected orinput for training and testing. At step 94, the collected images aresegmented by segmenter 22 so as to generate segmentation maps (asdescribed above).

At step 96, the image data 38 and non-image data 40 are annotated withlabels provided by qualified experts with domain knowledge. In a medicalapplication, the training classification outputs may be determined basedon subject clinical records. For example, if the classification outputis to include a fracture probability score, then the training output isthe timing (post-scan) of a fracture, ascertained from the subject'smedical history—and, where no fracture is apparent, recorded as“negligible risk” (or some equivalent designation). If theclassification output is to include a score for categories thatcorrespond to known medical conditions, then the training output is theactual medical condition of the subject, also ascertained from thesubject's clinical records.

At step 98, the data (comprising segmentation maps 50 and non-image data40) is split into a training set, a development or ‘dev’ set (which maybe omitted in some implementations), and a testing set, each for adifferent use. The training set is the data on which the learningalgorithm is to be run; the dev set is the data used to tune theparameters; the testing set is the data to be used to evaluate theperformance of the trained model.

At step 100, the layers of the deep neural network are implemented. Eachlayer consists of artificial neurons. An artificial neuron is amathematical function that receives one or more inputs and sums them toproduce an output. Usually, each input is separately weighted, and thesum is passed through a non-linear function. As the neural networklearns, the weights of the model are adjusted in response to the error(the difference between the network output and the annotations) itproduces until the error cannot be reduced further.

At step 102, the parameters of classification neural network58—including layer numbers, neuron numbers, neuron weights, and neuronfunction parameters, etc.—are initialized. At step 104, the learningalgorithm runs on the training data set to update the parameters ofclassification neural network 58. For example, the parameters might beupdated by determining a gradient of a loss function. The loss functionis calculated by the labelled classification output and output generatedby classification neural network 58. The dev data may be used toinvestigate the learning procedure and tune the layers and parameters.

At step 106, classifier 30—provided with classification neural network58—is tested on the testing data. At step 108, an evaluation is made asto whether the performance of classifier 30 is satisfactory. If theperformance is unsatisfactory, processing returns to step 92 where moretraining data is collected.

If, at step 108, the performance is found to be satisfactory, processingcontinues at step 110, where the trained classifier 30 is outputted orflagged for deployment, or released for use. Processing then ends.

It will be understood to persons skilled in the art of the inventionthat many modifications may be made without departing from the scope ofthe invention, in particular it will be apparent that certain featuresof embodiments of the invention can be employed to form furtherembodiments.

It is to be understood that, if any prior art is referred to herein,such reference does not constitute an admission that the prior art formsa part of the common general knowledge in the art in any country.

In the claims which follow and in the preceding description of theinvention, except where the context requires otherwise due to expresslanguage or necessary implication, the word “comprise” or variationssuch as “comprises” or “comprising” is used in an inclusive sense, i.e.to specify the presence of the stated features but not to preclude thepresence or addition of further features in various embodiments of theinvention.

1. A system for classifying a structure or material in an image of asubject, comprising: a segmenter configured to segment an image into oneor more segmentations that correspond to respective structures ormaterials in the image, and to generate from the segmentations one ormore segmentation maps of the image including categorizations of pixelsor voxels of the segmentation maps assigned from one or more respectivepredefined sets of categories; a classifier that implements a trainedclassification machine learning model configured to generate, based onthe segmentations maps, one or more classifications and to assign to theclassifications respective scores indicative of a likelihood that thestructure or material, or the subject, falls into the respectiveclassifications; and an output for outputting a result indicative of theclassifications and scores.
 2. A system as claimed in claim 1, whereinthe classifier generates the one or more classifications based on thesegmentations maps and non-image data pertaining to the subject.
 3. Asystem as claimed in claim 1, wherein the segmenter comprises: i) astructure segmenter configured to generate structure segmentation mapsincluding categorizations of the pixels or voxels assigned from apredefined set of structure categories, ii) a material segmenterconfigured to generate material segmentation maps includingcategorizations of the pixels or voxels assigned from a predefined setof material categories, and/or iii) an abnormality segmenter configuredto generate abnormality segmentation maps including categorizations ofthe pixels or voxels assigned from a predefined set of abnormality ornormality categories.
 4. A system as claimed in claim 3, wherein thestructure segmenter is configured to employ a structure segmentationmachine learning model to generate the structure segmentation maps, thematerial segmenter is configured to employ a material segmentationmachine learning model to generate the material segmentation maps, andthe abnormality segmenter is configured to employ an abnormalitysegmentation model to generate the abnormality segmentation maps.
 5. Asystem as claimed in claim 1, further comprising a segmentation mapprocessor configured to down-sample or otherwise process thesegmentation maps before the segmentation maps are input by theclassifier.
 6. A system as claimed in claim 1, wherein theclassification machine learning model comprises (a) a neural network, asupport vector machine, and/or a decision tree; or (b) a neural networkthat includes convolutional neural network layers and fully-connectedneural network layers.
 7. A system as claimed in claim 1, wherein theimage is a medical image, and the classifications correspond to (i)probabilities that the structure or material, or the subject, willsustain a specified condition or symptom in respective timeframes; (ii)probabilities that the structure or material, or the subject, willsustain a specified condition or symptom in respective timeframes thatinclude a shorter-term timeframe, a longer-term timeframe, and at leastone intermediate-term timeframe intermediate the shorter-term timeframeand the longer-term timeframe; (iii) probabilities that the structure ormaterial, or the subject, will sustain respective conditions orsymptoms; (iv) probabilities of respective rates of disease or pathologyprogression; (v) probabilities of respective rates of disease orpathology progression, the classifications comprising classificationscorresponding any one or more of: stable, modest deterioration, andaccelerated deterioration; (vi) probabilities of efficacy of respectivetreatment options; (vii) probabilities of efficacy of respectivetreatment options, the treatment options including an antiresorptivetreatment and/or an anabolic treatment; (viii) respective medicalconditions; and/or (ix) respective medical conditions that include anyone or more of: osteomalacia, tumour, osteonecrosis and infection. 8.(canceled)
 9. A system as claimed in claim 1, wherein the trainedclassification machine learning model is a model trained with image dataand non-image data relating to training subjects, and generates therespective scores based on image data and non-image data relating to thesubject.
 10. A computer-implemented method for classifying a structureor material in an image of a subject, comprising: segmenting an imageinto one or more segmentations that correspond to respective structuresor materials in the image; generating from the segmentations one or moresegmentation maps of the image including categorizations of pixels orvoxels of the segmentation maps assigned from respective predefined setsof categories of the structure or material; using a trainedclassification machine learning model to generate, based on thesegmentations maps, one or more classifications and to assign to theclassifications respective scores indicative of a likelihood that thestructure or material, or the subject, falls into the respectiveclassifications; and outputting a result indicative of theclassifications and scores.
 11. A method as claimed in claim 10, whereinthe trained classification machine learning model generates (i) the oneor more classifications based on the segmentations maps and non-imagedata pertaining to the subject, and/or (ii) the respective scores basedon image data and non-image data relating to the subject having beentrained with image data and non-image data relating to trainingsubjects.
 12. A method as claimed in claim 10, wherein forming the oneor more segmentations comprises: i) generating structure segmentationmaps including categorizations of the pixels or voxels assigned from apredefined set of structure categories, ii) generating materialsegmentation maps including categorizations of the pixels or voxelsassigned from a predefined set of material categories, and/or iii)generating abnormality segmentation maps including categorizations ofthe pixels or voxels assigned from a predefined set of abnormality ornormality categories.
 13. A method as claimed in claim 12, includingemploying a structure segmentation machine learning model to generatethe structure segmentation maps, a material segmentation machinelearning model to generate the material segmentation maps, and anabnormality segmentation model to generate the abnormality segmentationmaps.
 14. A computer-implemented method for training a classificationmachine learning model for classifying a structure or material in animage of a subject, the method comprising: dividing annotatedsegmentation maps and annotated non-image data into a training set and atesting set, the annotated segmentation maps obtained by segmenting eachof one or more images into one or more segmentations that correspond torespective structures or materials in the respective image; (a)implementing a classification machine learning model, includinginitializing parameters of the classification machine learning model;(b) updating the parameters of the classification machine learning modelby running a learning algorithm on the training data; (c) testing theclassification machine learning model on the testing data; (d)evaluating whether the classification machine learning model hassatisfactory performance; repeating steps (a) to (d) when theperformance is found in step (d) not to be satisfactory; and outputtingthe classification machine learning model for deployment as a trainedclassification machine learning model, or flagging the classificationmachine learning model as a trained classification machine learningmodel.
 15. A method as claimed in claim 14, including, when theperformance is found in step (d) not to be satisfactory, receiving andusing more image and non-image data for training the classificationmachine learning model.
 16. A method as claimed in claim 14, wherein theclassification machine learning model comprises a neural network havinga plurality of layers comprising artificial neurons, wherein theparameters comprise layer numbers, neuron numbers, neuron weights, andneuron function parameters; and testing the classification machinelearning model includes testing the classification machine learningmodel on the testing data.
 17. A method as claimed in claim 14,including dividing the annotated segmentation maps and the annotatednon-image data into the training set, a development set and the testingset, and using the development data to investigate the learningprocedure and to tune the parameters.
 18. A computer-implementeddiagnostic method, comprising the method of claim
 10. 19. A computerprogram, comprising program code configured, when executed by one ofmore computing devices, to implement+ed the method of claim
 10. 20. Acomputer-readable medium, comprising the computer program of claim 19.21. A classification machine learning model trained according to themethod of claim
 14. 22. A method as claimed in claim 14, wherein theimage is a medical image, and the classification machine learning modelis configured to generate one or more classifications that correspond to(i) probabilities that the structure or material, or the subject, willsustain a specified condition or symptom in respective timeframes; (ii)probabilities that the structure or material, or the subject, willsustain a specified condition or symptom in respective timeframes thatinclude a shorter-term timeframe, a longer-term timeframe, and at leastone intermediate-term timeframe intermediate the shorter-term timeframeand the longer-term timeframe; (iii) probabilities that the structure ormaterial, or the subject, will sustain respective conditions orsymptoms; (iv) probabilities of respective rates of disease or pathologyprogression; (v) probabilities of respective rates of disease orpathology progression, the classifications comprising classificationscorresponding any one or more of: stable, modest deterioration, andaccelerated deterioration; (vi) probabilities of efficacy of respectivetreatment options; (vii) probabilities of efficacy of respectivetreatment options, the treatment options including an antiresorptivetreatment and/or an anabolic treatment; (viii) respective medicalconditions; and/or (ix) respective medical conditions that include anyone or more of: osteomalacia, tumour, osteonecrosis and infection.
 23. Asystem for training a classification machine learning model forclassifying a structure or material in an image of a subject, the systemcomprising a processor configured to: divide annotated segmentation mapsand annotated non-image data into a training set and a testing set, theannotated segmentation maps obtained by segmenting each of one or moreimages into one or more segmentations that correspond to respectivestructures or materials in the respective image; (a) implement aclassification machine learning model, including initializing parametersof the classification machine learning model; (b) update the parametersof the classification machine learning model by running a learningalgorithm on the training data; (c) test the classification machinelearning model on the testing data; (d) evaluate whether theclassification machine learning model has satisfactory performance;repeat steps (a) to (d) when the performance is found in step (d) not tobe satisfactory; and output the classification machine learning modelfor deployment as a trained classification machine learning model, orflag the classification machine learning model as a trainedclassification machine learning model.
 24. A classification of astructure or material in an image of a subject, generated according tothe method of claim 10.